Skip to content

Lower Nx.top_k (Nx.Block.TopK) in the native Expr compiler - #187

Merged
ausimian merged 1 commit into
feat/expr-compilerfrom
feat/expr-compiler-topk
Jun 6, 2026
Merged

Lower Nx.top_k (Nx.Block.TopK) in the native Expr compiler#187
ausimian merged 1 commit into
feat/expr-compilerfrom
feat/expr-compiler-topk

Conversation

@ausimian

@ausimian ausimian commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Nx.top_k now compiles under native instead of raising "does not yet lower
the block Nx.Block.TopK"
. With this, all 8 example livebooks lower fully
native
modernbert_classification was the lone holdout (found by the
forced-native livebook probe).

What it is

Nx.Block.TopK is a multi-output block ({values, indices}), and
Emily.Backend has no top_k override (mx::topk yields values only, not
the indices Nx's contract requires). So the evaluator computes it via the
block's default expansion: argsort(desc) → take_along_axis → slice the top k. Every op there already lowers, so the compiler lowers that same
expansion
and projects the two leaves via :elem (the {:multi_refs, …}
machinery the tuple-cond fix added) — bit-identical to the evaluator.

The subtle part — binding the block's parameters

Nx.Defn.Expr.expr_block builds a block's expansion against fresh
per-position :parameter nodes
, not the real in_args. So lowering the
expansion directly makes a block parameter 0 fall through to the outer
function's {:input, 0}
— a completely different tensor. That's exactly how
ModernBERT failed: its outer input 0 is the 1-D input_ids, so top_k's
argsort ran on a 1-D array → "invalid axis 1 for array with 1 dimensions"
at replay, even though logits/softmax were correct {1, num_labels}.

The fix binds the block parameters to the lowered in_args (the same thing
while does via param_seed) before lowering the expansion. Eval never tripped
on this because it evaluates the bound expansion directly on real tensors.

Changes

  • ir.exlower_block for Nx.Block.TopK: lower the in_args, seed the
    cache so each block :parameter resolves to its arg, lower the expansion
    leaves, return {:multi_refs, [vref, iref]}. collect_block_params/2 walks
    the (small) expansion to find the parameters.
  • compiler_equivalence_test.exs — values + s32 indices match; a
    downstream op consuming both leaves; and a regression case where outer
    input 0 is 1-D (the ModernBERT shape), which an unbound parameter would crash.

Verification

  • mix precommit green: 694 tests, 0 failures.
  • The modernbert_classification livebook, forced to native: true, native_fallback: :raise, now runs end-to-end native (was the 1 failing of
    8). The other 7 were already native.

Nx.top_k now compiles under native instead of raising "does not yet
lower the block Nx.Block.TopK". It's a multi-output block {values,
indices}: Emily.Backend has no top_k override (mx::topk yields values
only, not the indices Nx's contract requires), so the evaluator computes
it via the block's default expansion (argsort desc -> take_along_axis ->
slice the top k).

The compiler lowers that same expansion and projects the two leaves via
:elem (the {:multi_refs, ...} machinery the tuple-cond path added).
Crucially, Nx.Defn.Expr.expr_block builds the expansion against FRESH
per-position :parameter nodes, not the real in_args, so the parameters
must be BOUND to the lowered in_args (the same binding `while` does via
param_seed). Without that, a block parameter falls through to the OUTER
function's {:input, pos} — a different tensor — which is exactly how
modernbert_classification hit "argsort axis 1 on a 1-D array" (its outer
input 0 is the 1-D input_ids). Result is bit-identical to the evaluator,
which runs the same bound expansion.

- ir.ex: lower_block clause for Nx.Block.TopK — lower the in_args, seed
  the cache so each block :parameter resolves to its arg, lower the
  expansion leaves, return {:multi_refs, [vref, iref]}.
  collect_block_params/2 walks the (small) expansion to find them.
- compiler_equivalence_test.exs: values + s32 indices match; a downstream
  op consuming both leaves; and a regression case where outer input 0 is
  1-D, so an unbound parameter would crash (the ModernBERT shape).

Closes the last gap from the forced-native livebook probe: all 8 example
livebooks now lower fully native (modernbert_classification was the
holdout, on top_k).
@ausimian
ausimian force-pushed the feat/expr-compiler-topk branch from a3e1217 to d95fff5 Compare June 6, 2026 07:47
@ausimian
ausimian merged commit 6406d83 into feat/expr-compiler Jun 6, 2026
3 checks passed
@ausimian
ausimian deleted the feat/expr-compiler-topk branch June 6, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant